Detection and Remediation of Stagnation in the Nelder--Mead Algorithm Using a Sufficient Decrease Condition

The Nelder-Mead algorithm can stagnate and converge to a non-optimal point, even for very simple problems. In this note we propose a test for sufficient decrease which, if passed for the entire iteration, will guarantee convergence of the Nelder-Mead iteration to a stationary point if the objective function is smooth. Failure of this condition is an indicator of potential stagnation. As a remedy we propose a new step, which we call an oriented restart, which reinitializes the simplex to a smaller one with orthogonal edges which contains an approximate steepest descent step from the current best point. We also give results that apply when objective function is a low-amplitude perturbation of a smooth function. We illustrate our results with some numerical examples.